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ABSTRACT 



A holistic approach to evaluating an adult education and 
training program or service was proposed that is based on consideration of 
the following: ethics, objectives, stakeholders, coverage, effects, and 
costs. According to the proposed approach, the following are among the design 
features of multimedia instruction to be considered before an evaluation: 
particular mode of technology-assisted learning; method of using technology 
in the instructional process; social aspects of the instructional process; 
relationship with other components of training/instruction; hardware 
requirements; software aspects; assumed learning sequences; medium of 
presentation of details; extent of interactivity, self -pacing, and 
self-direction; and subject matter interest. The actual evaluation procedure 
is a six- step process during which evaluators evaluate a program against the 
prescribed criteria to ascertain the following: whether the program or 
service is ethical; the extent to which the program or service covers those 
who are most in need; the costs, benefits, and utilities of the program or 
service; whether the program or service achieved its key objective (s) ; the 
net effect of the program or service; and the extent to which the 
perspectives and interests of all stakeholders have been considered and met. 
(24 endnotes) (MN) 
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EVALUATING THE EFFECTIVENESS OF 
MULTIMEDIA BASED LEARNING 




James A Athanasou 
Faculty of Education 
University of T echnology, Sydney 



There are major paradigm shifts occurring in the delivery of formal instruction with 
new pathways and options for learning at all stages of human development. Typical of these 
developments are: (a) the Ford factory in Cologne, which uses flexible learning through 
tutorials, simulations, hypermedia to assist workers obtain vocational qualifications ; or (b) 
using optical disk-based technology for delivering courseware in the learning of cell biology" 2 ; 
or (c)Telescopia, a technology-driven European Union project for distance learning 3 ; or (d) 
Just in time Open Learning which combines computer-mediated communications with a 
hypermedia database 4 ; or (e) the introduction of open and flexible learning in three British 
Training and Enterprise Councils, which reduced unit costs by 10-50%. 5 Such innovations, 
when coupled with the power of multimedia based learning, have major implications for 
educational evaluation. 

Although we may pride ourselves on the ability to act rationally, it is not always the 
case that our educational programs or training services are reviewed objectively or 
adequately. For instance, despite the fact that Australian training expenditure in 1996 
accounted for some 2.5% of gross wages, only 28.6% of in-house courses had their content 
and subject matter decided on the basis of earlier course evaluations or even a formal 
training needs analysis (32%) 5 Yet at the same time there are many trainers who wish to be 
self-critical and are conscious of the need for formal evaluations. Accordingly, one of the 
most frequently asked questions that I encounter is: 'How should I evaluate industrial and 
commercial training?’ 
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Unfortunately there is a common view that almost any educational program is 
beneficial. Well it might be but it will be more certain when we choose to compare it to 
another program or to a set of standards. By way of introduction, let me cite examples from 
widely divergent fields to further emphasise the need for evaluation as a component of all 
learning programs: health services training; computer-based instruction; classroom 
instruction; and management in commerce and industry. 

Over the last 10 years Congress provided nearly $(A) 2.6 billion for health 
professions training, including nurses, nurse practitioners and nurse midwives. Six of the 23 
programs had never been evaluated and the evaluations of the remaining 17 were unable to 
be generalised beyond their locality to determine any national impact 7 . This scenario is not 
atypical. A major Australian banking corporation introduced training through an interactive 
CD-ROM package (which typically requires about 600 hours of development/production 
time). It was designed to provide flexible learning throughout its branch network on the basis 
of reducing the travel costs associated with staff training. The costly computer-based 
instruction was not evaluated by the developers (other than software testing) because it was 
considered to be effective but independent assessments indicated no reduction in face-to- 
face instruction, an outright preference for using written manuals of instruction and in many 
cases the CD-ROM package was not even tried. Thirdly, comparisons of computer-assisted 
instruction versus conventional teaching on 5,379 Job Corps trainees 8 found small 
achievement gains even after some 60 hours of instruction. Such comparisons are a useful 
start but they are fraught with difficulties because we know that both learner background 
(e.g., prior knowledge, interest) as well as instructional characteristics (e.g., design, level of 
control, navigation) affect the evaluation of multimedia learning 9 Fourthly, an estimated 5(A) 
40 billion is spent in the US on management training, yet a recent survey indicated that this 
was not being evaluated. It is not the case that trainers are reluctant to evaluate but what are 
some of the barriers? Some of the reasons centred on a lack of standards, the difficulty in 
measuring learning, insufficient staff, difficulties in quantifying or measuring evaluations, as 
well as a lack of expertise in evaluation methods. 10 . 




Figure 1 . The foundations of educational evaluation 



Evaluation as a way of deciding worth and merit 

The introduction of any technology-assisted learning program or service reflects a 
personal, social and organisational investment that needs to be reviewed. How can one be a 
professional if there is no built-in monitoring or some form of program evaluation? Without 
this, we can end up with ‘sloppy thinking', with poor judgements about the efficacy of our 
services and below average performance. 

Evaluation can serve as a useful component of all programs and services, ideally in 
a formative sense to make amendments and improvements as well as a general monitoring 
role. The Joint Committee on Standards for Educational Evaluation defines the way in which 
I am using the term evaluation, namely, as 'the systematic investigation of the worth or merit 
of an object 11 . 

In a field such as education or training, which has a proud tradition of theory and 
practice, there could be nothing more consistent with professionalism than to mandate the 
evaluation of major programs and services, especially technology-assisted learning because 
of its potential to change ways of learning and instruction. Evaluation was developed largely 



by educators and has an applied emphasis. It has been refined over many years to become 
a separate discipline and area of study in its own right, growing from developments in social 
science research methodologies and the healthy respect that we have for evidence and 
facts. 

For the purposes of this paper I am concerned with public, structured, formal and 
summative evaluations and I shall focus on the key steps for evaluating a program or 
service, mainly in the area of adult education and training 12 . The aim of this paper is to 
outline a holistic or synthetic approach that you can use to evaluate training and educational 
programs (see figure). I do not dwell on the specific plans or procedures (ie, methodology) 
but focus on six key issues. 



ETHICS 




Figure 2. The ECCOES framework for evaluations 

Where available I shall refer to examples from technology-assisted learning and at 
the outset I wish to state that I do not represent any particular program, product or service. I 
deal mainly with computer-assisted learning 13 as part of a case study for evaluation even 
though this is only one aspect of multimedia learning. Some of the examples that I am using 
will come from other fields of education and training but the ideas can readily be transferred 
to all areas of multimedia learning within industrial and commercial training (see also the 
table below dealing with design features). 
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Steps in an evaluation of an education or training program 

Let me now direct your attention to the criteria that I think should be used in an 
evaluation. An attempt has been made to provide you with a framework and a structure for 
your decision making. Six steps are listed and I have called this a holistic (or synthetic) 
approach to evaluation. It is up to you, however, to decide how to combine the answers to 



these six bits of information in deciding upon the merit or worthiness of any program or 
service (ie, technology-assisted learning). 

Step 1 - Is the program or service ethical? 

This is the first and foremost question prior to any program evaluation. It takes into 
account, questions such as: 

• Ethics/morality, 

• Social and political implications, 

• Legality, 

• Any harmful ecological issues, 

• Any impact on privacy or confidentiality, 

• Any abuse of privilege, and 

• Whether human ethical guidelines have been satisfied. 

I would not presume to tell you what is appropriate but your own professional and ethical 
codes might serve as a guide. You may wish to question the ways in which the technology is 
applied to learning or the rationale for the introduction of technology-assisted learning. The 
latter should probably involve valid social, economic, vocational or pedagogic reasons. 

Step 2 - To what extent does the programs or services cover those who are most in 
need? 

The second issue here is the coverage of the program (eg, coverage efficiency, 
specificity, and sensitivity). While it is important to establish that large numbers of people are 
capable of being served it is also important to relate it to questions of: 

• The incidence 14 of the educational or training need, 

• The prevalence of the educational or training need, 

• The numbers of people who are at risk 

• The sensitivity of the program 

• The specificity of the education or training 

• The attendance and completion rates for any course. 

A helpful calculation is to determine the coverage efficiency 15 of a program. 

Number in Number not- 

need served in-need served 

Coverage efficiency = 1 00 x 



Total number Total number 

in need served 

An example, would be an organisation with 150 staff that was trained in new techniques but 
25 already had the required skills. The coverage efficiency of this training program involving 
face-to-face instruction was a minus 80% (125/125 - 25/150). 

Step 3 -What are the costs, benefits and utilities of the program or service? 

Most education and training programs are labour intensive in delivery and not always 
as cost-effective as we would like them to be. My initial estimates of the general break-even 
cost of tertiary level face-to-face classroom instruction or training (without additional 
resources) is around $35 per person per hour or around $525 per hour for a group of 15 (not 
inclusive of participant salaries or wages) or around $14,000 per group for many semester 
length units or modules. This can compare favourably with more flexible forms of delivery 
where the bulk of the costs are transferred to the production stages rather than to delivery. It 
can be useful to consider the ratio of production time to delivery or learning; in the area of 
computer-assisted learning the ratio of production time to one hour of instruction has varied 
from 30 to as much as 1,000 hours. Nevertheless, the available evidence indicates that the 
overall reduction in instructional time for computer-assisted learning has averaged out at 
around one-third. 



It is a helpful first step to take into account all the likely costs associated with the 
production and delivery of a program. These include salaries, on-costs, participant wages, 
rental and all the other overheads that would be required if the program was to be developed 
or offered independently or externally. Many of our development and delivery costs are 
“buried” and the real expenses are hardly ever considered. 

Various cost-benefit analyses allow us to compare different alternatives when their 
inputs and outcomes are measured in dollar terms. If money values have not been assigned 
to inpu ts and outputs then these can be determined in some fashion. 

Cost effectiveness = cost of program / effect of program on a common measure 



There are many different types of cost analyses. Cost feasibility is straightforward and 
answers the question of whether we have sufficient funds or resources. Cost effectiveness 
looks at the cost per unit and an example of this is to calculate the return on training 
investment. For example, Federal Express reported a 24% return on training investment over 
a period of three months for a program which gave 20 new employees training in driving 
company vans. Finance and engineering specialists allocated dollar values to the 
performance categories and performance was compared to an "untrained" control group 16 . 
Cost-benefit analyses can be used to contrast the proposed advantages and disadvantages 
and a cost-utility analysis can be used routinely to make comparisons in the ultimate value 
of what you are trying to achieve. 



Step 4 - Did the program or service achieve its key objective(s)? 

The focus of this question is to concentrate your attention on the key objectives of 
your program. To what extent were you able to achieve these outcomes? Here we are 
focusing on the gross outcomes or effects of the program, 

Surveys of training and education experiences have indicated that 63% of in-house 
and external training programs in Australia failed even to assess learning. Some say that 
you do not really need to assess learning because it is obvious. Well, official figures 
indicated that some 400,000 of the 3 million people who undertook training programs in 
Australia reported that training had not improved job performance. It may have been a 
satisfactory experience and learning may have occurred (either short-term or long-term); but 
sadly, we shall never know 17 . 

A peculiar difficulty in educational evaluations is separating out the short- from the 
long-term effects of education. You may have had the experience that some education and 
training programs are evaluated positively by participants but have few long-term effects 
while others might not be evaluated well in the short-term but have significant long-term 
effects. This placebo effect of any training program should not be overlooked. 

Evaluations of computer-assisted learning across some 180 studies 18 indicated that 
average performance in examinations is raised by around 11 percentile points and affected 
attitudes to instruction, towards computers but not towards subject matter. 

A multi-dimensional approach that offers a straightforward, pragmatic and initial basis 
for an educational evaluation is Kirkpatrick’s hierarchy of evaluation 19 . This is a 
comprehensive approach to evaluation that has been around for many years and which is 
- popular in adult education and training. It focuses on four levels of program outcome and is 
illustrated in the figure below. It is especially suited to training contexts where specific 
outputs are of interest right from the outset. It is a pity, therefore, that the four levels of 
program outcome are not used more often. 



REACTION 



LEARNING 



BEHAVIOUR 



RESULTS 



An evaluation hierarchy 

I. Reaction - eg, program participants' estimates of satisfaction provide an 
immediate level of evaluation 

II. Learning - eg, the extent of learning (ie, skill acquisition, attitude change) that 
has been achieved is also of interest 

III. Behaviour - eg, the extent to which learning has generalised to the work 
situation or there has been a transfer of skills is assessed at this level through 
follow-up after a program 

IV. Results - eg, the wider impact of a program in the community or the 
organisation is observed at this level in the hierarchy. 

Most people undertake the evaluations of courses only at Level 1 through the use of 
end-of-course questionnaires. This is helpful but is only part of the picture. At the very least, 
the questionnaires need to be standardised, comparable with other databases of results, 
anonymous and processed independently. My opinion is that all course participants should 
be assessed at level 1. Depending on the size of the group, course participants could 
evaluate either (a) the quality of the training or instruction; (b) the curriculum or content; or (c) 
the overall course. Further random samples should be taken to assess the effect of training 
at Levels 2 (say 80% of course participants over time where numbers are less than 300), 
with smaller group samples at Levels 3 and 4. In conducting these surveys I recommend the 
benefits of matrix sampling (ie, asking different questions to individuals and combining 
results for the group) rather than asking every person every question. 

The simplicity of the model masks some real complexities in evaluating the extent to 
which skills, knowledge and attitudes have been acquired, retained and maintained. You 
should also be aware that there are criticisms of this approach 20 but it has served as an 
influential catalyst for training evaluation. 

Step 5 - What is the net effect of the program or service? 

A challenge for any evaluator is to determine the effectiveness of his/her program or 
service and to contrast this with comparable programs. This means that we need to be able 
to describe the impact of a program, its significance and the size of any learning effect. 
Firstly, the net outcome must be established and it is not always the case that significant 
differences in test scores are evident when multimedia is compared with traditional 
instruction, for example in university tutorials on telecommunications, 2 but may be evident in 
other characteristics such as attitudes,„One..oTthe advantages that have been claimed for 
multimedia in language instruction are its ability to facilitate learning but also enhance 
student attitudes or motivation to learn. 22 

We also might want to analyse the differential impact that training is having for 
various groups and some analytical framework is helpful. I would refer you to almost any 
textbook on research designs for some help in selecting the most appropriate method of 
analysis or design. Without comparisons, however, it is impossible to make any statement 
about value. This is because program evaluation is largely a relative process. 

Then again, another factor in making comparisons is how training can interact with 
the pre-existing skill level and expertise of the participants. An excellent example from one of 
my graduate students might help here. Three groups of nurses (community nurses, ward 
staff, new graduates) had largely identical scores on a pre-test knowledge of drug and 
alcohol. Following training the follow-up assessments after some 2-3 months indicated the 
greatest training gains in knowledge (on a 25-item test) for the most experienced nursing 
staff (F(2,42) = 10.2, p<0.0002). There were significant gains in training but these varied 
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across the levels of experience. This type of outcome would be familiar to experienced 
trainers and it highlights the need to make comparisons. 



GROUP lllllll 


.Pre-test !j| 

: :jyteiairv: SCO 


Post-test 


New graduates 


13(4) 


16(3) 


Ward staff 


14(3) 


19(3) 


Community nurses 


14 (4) 


20 (2) 



Note. All scores rounded 

□ 



□ 

□ 

Step 6 - To what extent have the perspectives and interests of all stakeholders been 
considered and met? 

Although our programs and services may be well-intentioned they may overlook the 
interests or needs of client groups. For instance, Medtronic is a company that operates in the 
area of implanted heart devices; it uses multimedia to increase learning and reduce training 
time but it found that the introduction of multimedia had to overcome resistance from its 
sales representatives 23 . 

The value of client consultation cannot be overemphasised because it alerts you to 
factors that you may not have considered important. Stakeholder perspectives are many and 
varied (eg, participants, staff, management, community, government) and some of the 
questions that you may wish to consider, could include 

• Why was the program, policy or initiative introduced? 

• What are its intended educational outcomes? 

• How do people feel about the program? 

• Who are the key stakeholders? 

• What are the social costs and benefits of the program? 

• What constraints are operating? 

• How will the evaluation findings be used? 

It should not be a surprise that corporations with the best training evaluation process (eg. 
IBM, Motorola University, Arthur Andersen, Florida Power, and AT&Ts School of Business) 
have a customer focus 24 . They determined client requirements: ‘Well over half of what IBM 
identified as customer requirements had not been previously measured by the training unif. 
The term “customers” relates directly to stakeholder perspectives and included participants, 
their managers, external customers, training councils, vendors, regulatory agencies, and 
upper-level management. 



Conclusions 

These six questions synthesise the educational evaluation process for me and I 
hope that they provide a brief but comprehensive framework within which you can operate. I 
think that they cover the most important issues, when you are ready to undertake an 
evaluation of any program or service. The six steps and some of the criteria are summarised 
in the checklist below. These steps, however, do not provide an automatic answer. 

The results form the basis for a decision about the worthiness of the program or 
service. With all the information from these six steps, you then have to decide whether the 
program has merit. In some cases a program may be costly but very effective; or it may be 
effective in the short-term and well-liked but may not be meeting long-term needs; or it may 
be less than perfectly ethical but have wide coverage and popularity; and so on. You will 
bring to the evidence a set of priorities, which must ultimately influence any decision. That is 
fine and it is exactly the way it should be - there is no 'right answer. 
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Some criteria will be more relevant to you than others but all need to be weighted 
appropriately for your decision. Such decisions involve complex judgements about each 
educational program or service. Maybe the best that we can hope for is to provide 
information that confirms our ideas about the potential of a program or that refutes claims 
about the value of a program. In this way educational and training evaluations will add to our 
store of knowledge about what works and what does not work in education and training and 
how well it works. 

To return to the perceived barriers for evaluation, firstly it is clear that evaluation does 
require expertise but that it can be acquired. It is not overly complex and is an inter- 
disciplinary field in which I would encourage your greater involvement (eg. through your own 
profession or specialist organisations such as the Australasian Evaluation Society). 
Secondly, many of the difficulties in evaluation, such as the lack of standards or the difficulty 
in isolating behaviours often relate to the ways in which programs were established and 
conceived at the outset. You may need to set your own standards and define the behaviours 
that you consider most important. Thirdly, the difficulties in measurement can be overcome 
by having well-stated and specific learning outcomes which are few in number. Qualitative 
training outcomes can be described using case studies or what we call ethnographic 
approaches that yield valuable descriptions. Fourthly, the issue of insufficient staff might be 
considered from the viewpoint of building evaluation into every training component from the 
outset. The easiest evaluations to conduct are those in organisations that keep excellent 
records or have set up mechanisms for program monitoring. It is not possible to make 
education or training perfect but overtime it is possible to reduce our errors through practical 
training evaluations and to improve gradually the quality of our professional efforts. 
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Table 2 Checklist for holistic evaluations of .training services , 

Issue 1 - Is the program or service ethical? 

Ethics/morality 

Social and political implications 
Legality 

Any harmful ecological issues 
Any impact on privacy or confidentiality 
Any abuse of privilege 
Reasons for the evaluation 

Whether human ethical guidelines have been satisfied 

Whether evaluation standards have been satisfied 

Issue 2 - To what extent does the program or service cover those who are most in need? 

The incidence of the educational or training need 

The numbers of people who are at risk 

The prevalence of the educational or training need 

The numbers of people who are at risk 

The sensitivity of the program 

The specificity of the education or training 

The attendance and completion rates for any course 

Issue 3 - What are the costs, benefits and utilities of the program or service? 

Cost feasibility 
Cost effectiveness 
Cost-utility 

Cost-benefit 

Issue 4 - Did the program or service achieve its key objective(s)? 

Reactions 

Learning 

Behaviours 

Results 

Issue 5 “ What is the net effect of the program or service? 

Comparison with other programs 
Comparison with pre-determined standards 

Determine statistical effect size (if relevant) 

Issue 6 “ To what extent have the perspectives and interests of all stakeholders been 
considered and met? 

Why was the program, policy or initiative introduced? 

What are its intended educational outcomes? 

How do people feel about the program? 

Who are the key stakeholders? 

What are the social costs and benefits of the program? 

What constraints are operating? 

What aspects will influence decision-making? 

How will these evaluation findings be used? 

Source: Adapted from Athanasou (1998a), p. 102 
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